HPC has long been a focus of both academic research and commercial development, and the field presents a bewildering array of standards, products, tools, and consortia. Any attempt at comparative analysis is complicated by the fact that many of these interrelate not as mutually exclusive alternatives, but as complementary component or overlapping standards.
Probably the most familiar, and certainly the oldest, approach is based on dedicated supercomputing hardware. The earliest supercomputers included vector-based array processors, whose defining feature was the capability to perform numerical operations on very large data arrays, and other SIMD (Single-Instruction, Multiple-Data) architectures, which essentially performed an identical sequence of instructions on multiple datasets simultaneously. More recently, multiple-instruction architectures, and especially SMPs (Symmetric Multi-Processors), have tended to predominate, although the most powerful supercomputers generally combine features of both.
With dramatic improvements in the processing power and storage capacity of “commodity” hardware and burgeoning network bandwidth, much of the focus has shifted toward parallel computing based on loosely-coupled clusters of general-purpose processors, including clusters of network workstations. Indeed, many of the commercially available high-performance hardware platforms are essentially networks of more or less generic processors with access to shared memory and a high-speed, low latency communications bus. Moreover, many of the available tools and standards for developing parallel code are explicitly designed to present a uniform interface to both multi-processor hardware and network clusters. Despite this blurring around the edges, however, it is convenient to draw a broad dichotomy between conventional hardware and clustering solutions, and the discussion below is structured accordingly.
Conventional Hardware solutions
Typical commercial end-users faced with performance bottlenecks consider hardware solutions ranging from mid- to high-end SMP server configurations to true “supercomputers.” In practice, they often follow a tortuous, incremental migration path, as they purchase and outgrow successively more powerful hardware solutions.
The most obvious shortcoming of this approach is the visible, direct hardware cost, but even more important are the indirect costs of integration, development, administration, and maintenance. For example, manufacturers and resellers generally provide support at an annual rate equal to approximately 20–30% of the initial hardware cost. Moreover, the increase in physical infrastructure requirements and the administrative burden is much more than linear to the number of CPUs.
But by far the most important issue is that each incremental hardware migration necessitates a major redevelopment effort. Even when the upgrade retains the same operating system (e.g., from one Sun Solaris™ platform to another), most applications require substantial modification to take advantage of the capabilities of the new target architecture. For migrating from one operating system to another (e.g., from NT™ or Solaris™ to Irix™), the redevelopment cost is typically comparable to that of new development, but with the additional burden of establishing and maintaining an alternative development environment, installing and testing new tools, etc. Both development and administration require specialized skill sets and dedicated personnel.
In sum, other indirect costs often total 7 to 9× direct hardware costs, when personnel, time-to-market, and application redevelopment costs are taken into account.
Clusters, Grids, and Virtual Supercomputers
The basic idea of bundling together groups of general-purpose processors to attack large-scale computations has been around for a long time. Practical implementation efforts, primarily within academic computer science departments and government research laboratories, began in earnest in the early 1980s. Among the oldest and most widely recognized of these was the Linda project at Yale University, which resulted in a suite of libraries and tools for distributed parallel processing centered around a distributed, shared memory model.
More elaborate and at a somewhat higher level than Lnda, but similar in spirit, PVM (for Parallel Virtual Machine) provided a general mechanism-based on a standard API and messaging protocol for parallel computation over networks of general-purposes processors. More recently, MPI (the Message Passing Interface) has gained ground. Although they differ in many particulars, both are essentially standards that specify an API for developing parallel algorithms and the behavioral requirements for participating processors. By now, libraries provide access to the API from C and/or Fortran. Client implementations are available for nearly every operating system and hardware configuration.
Grid Computing represents a more amorphous and broad-reaching initiative—in certain respects, it is more a philosophical movement than an engineering project. The overarching objective of Grid Computing is to pool together heterogeneous resources of all types (e.g., storage, processors, instruments, displays, etc.), anywhere on the network, and make them available to all users. Key elements of this vision include decentralized control, shared data, and distributed, interactive collaboration.
A third stream of development within high-performance distributed computing is loosely characterized as “clustering.” Clusters provide HPC by aggregating commodity, off-the-shelf technology (COTS). By far the most prominent clustering initiative is Beowulf, a loose confederation of researchers and developers focused on clusters of Linux-based PCs. Another widely recognized project is Berkeley NOW (Network of Workstations), which has constructed a distributed supercomputer by linking together a heterogeneous collection of Unix and NT workstations over a high-speed switched network at the University of California.
There is considerable overlap among these approaches. For example, both Grid implementations and clusters frequently employ PVM, MPI, and/or other tools, many of which were developed initially to target dedicated parallel hardware. Nor is the terminology particularly well defined; there is no clear division between “grids” and “clusters,” and some authors draw a distinction between “clusters” or dedicated processors, as opposed to “NOWs” (Networks of Workstations), which enlist part-time or intermittently available resources.
Clusters and Grids as Enterprise Solutions
The vast majority of clusters and Grid implementations are deployed within large universities and Government research laboratories. These implementations were specifically developed as alternatives to dedicated supercomputing hardware, to address the kinds of research problems that formed the traditional domain of supercomputing. Consequently, much of the development has focused on emulating some of the more complex features of the parallel hardware that are essential to address these research problems.
The earliest commercial deployments also targeted traditional supercomputing applications. Examples include: hydrodynamics and fluid-flow, optics, and manufacturing process control. In both research and commercial settings, clustering technologies provide at least a partial solution for two of the most serious shortcomings of traditional supercomputing: (1) up-front hardware cost, and (2) chronic software obsolescence (since the system software to support distributed computing over loosely coupled networks must, out of necessity, provide substantial abstraction of the underlying hardware implementation).
However, clusters and grid implementations share, and in many cases, exacerbate, some of the most important weaknesses of supercomputing hardware solutions, particularly within a commercial enterprise environment. Complex, low-level APIs necessitate protracted, costly development and integration efforts. Administration, especially scheduling and management of distributed resources, is burdensome and expensive. In many cases, elaborate custom development is needed to provide fault tolerance and reliability. Both developers and administrators require extensive training and special skills. And although clusters offer some advantages versus dedicated hardware with respect to scale, fragility and administrative complexity effectively impose hard limits on the number of nodes—commercial installations with as many as 50 nodes are rare, and only a handful support more than 100.
These weaknesses have become increasingly apparent, as commercial deployments have moved beyond traditional supercomputing applications. Many of the most important commercial applications, including the vast majority of process-intensive financial applications, are “naturally parallel.” That is, the computation is readily partitioned into a number of more or less independent sub-computations. Within financial services, the two most common sources of natural parallelism are portfolios, which are partitioned by instrument or counterparty, and simulations, which are partitioned by sample point. For these applications, complex features to support process synchronization, distributed shared memory, and inter-process communication are irrelevant—a basic “compute server” or “task farm” provides the ideal solution. The features that are essential, especially for time-sensitive, business-critical applications, are fault-tolerance, reliability, and ease-of-use. Unnecessary complexity drives up development and administration costs, undermines reliability, and limits scale.
HPC in the Financial Services Industry
The history of HPC within financial services has been characterized by inappropriate technology. One of the earliest supercomputing applications on Wall Street was Monte Carlo valuation of mortgage-backed securities (MBS)—a prototypical example of “naturally parallel” computation. With deep pockets and an overwhelming need for computing power, the MBS trading groups adopted an obvious, well-established solution: supercomputing hardware, specifically MPPs (Massively Parallel Processors).
Although this approach solved the immediate problem, it was enormously inefficient. The MPP hardware that they purchased was developed for research applications with intricate inter-process synchronization and communication requirements, not for naturally parallel applications within a commercial enterprise. Consequently, it came loaded with complex features that were completely irrelevant for the Monte Carlo calculations that the MBS applications required, but failed to provide many of the turnkey administrative and reliability features that are typically associated with enterprise computing. Protracted in-house development efforts focused largely on customized middleware that had nothing to do with the specific application area and resulted in fragile implementations that imposed an enormous administrative burden. Growing portfolios and shrinking spreads continued to increase the demand for computing power, and MPP solutions wouldn't scale, so most of these development efforts have been repeated many times over.
As computing requirements have expanded throughout the enterprise, the same story has played out again and again—fixed-income and equity derivatives desks, global credit and market risk, treasury and Asset-Liability Management (ALM), etc., all have been locked in an accelerating cycle of hardware obsolescence and software redevelopment.
More recently, clustering and grid technologies have offered a partial solution, in that they reduce the upfront hardware cost and eliminate some of the redevelopment associated with incremental upgrades. But they continue to suffer from the same basic defect—as an outgrowth of traditional supercomputing, they are loaded with irrelevant features and low-level APIs that drive up cost and complexity, while failing to provide turnkey support for basic enterprise requirements like fault-tolerance and central administration.
The invention, as described below, provides an improved, Grid-like distributed computing system that addresses the practical needs of real-world commercial users, such as those in the financial services and energy industries.